Skip to content

docs(queries): Query limits & performance, LCQL reference, and compiled SDK snippets#294

Merged
tomaz-lc merged 18 commits into
masterfrom
docs-query-limits-performance
Jul 10, 2026
Merged

docs(queries): Query limits & performance, LCQL reference, and compiled SDK snippets#294
tomaz-lc merged 18 commits into
masterfrom
docs-query-limits-performance

Conversation

@tomaz-lc

@tomaz-lc tomaz-lc commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Details

This PR expands the Query Console / LCQL documentation under docs/4-data-queries/ and includes a CI/security hardening pass on the repository's GitHub Actions workflows. On the docs side it adds a new "Query Limits & Performance" page, greatly expands the LCQL examples and reference material, documents how to discover the per-organization Search API endpoint, refreshes and annotates the Query Console screenshot, and corrects several accuracy issues in the programmatic examples.

It also introduces mechanisms so the docs stay correct over time: custom syntax highlighting for LCQL code blocks, extraction of every Python and Go SDK example into standalone files that are compiled in CI so they cannot drift from the published SDKs, and a CI check that catches ordered-list numbering silently breaking in the rendered site.

What's included

New page - Query Limits & Performance (docs/4-data-queries/query-limits-and-performance.md):

  • Query types (stateless, projection, aggregation, stateful) and how each executes (paged vs whole-timeline).
  • Data streams (event, detection, audit).
  • Concurrent-query limits and the HTTP 429 behavior, query timeouts, and aggregation cardinality guidance.
  • Query progress and cost reporting (pre-flight estimate vs per-page actual billing), including a worked progress-bar example - the clamp(batchesCompleted / batchesInScope, 0, 100%) formula the Query Console uses - in Python, Go, and bash + jq.
  • A guide to writing efficient queries, plus a troubleshooting section.

LCQL Examples (docs/4-data-queries/lcql-examples.md):

  • New "Time Range" section explaining how the time range is supplied per interface (the query string for raw LCQL and Replay, the Console time picker, the Search API startTime/endTime parameters, and the CLI --start/--end flags) and the supported relative, absolute, and bounded formats.
  • New "Common Operators and Patterns" section, and expanded stateful operators (with child / with descendant / with events) with sample data, plus subtree-wildcard examples.
  • Correctness fixes (for example COUNT vs COUNT_UNIQUE on a GROUP BY key), added projections, and high-cardinality GROUP BY guidance.

Programmatic reference (docs/4-data-queries/index.md):

  • LCQL field paths corrected to / notation throughout.
  • Documents Search API endpoint discovery via /v1/orgs/{oid}/url with a per-region reference table, recommending discovery over hardcoding a direct endpoint.
  • Go SDK examples corrected to use the real SDK methods and the correct client initialization; CLI saved-query flags corrected.

Query Console UI (docs/4-data-queries/query-console-ui.md):

  • Replaced the overview screenshot with an annotated version (numbered UI elements) and fixed the numbered-list rendering (the sub-lists were flattening and the 1-10 list fragmented into restarting blocks; re-indented to 4-space nesting so it renders as one continuous list).
  • Documented two elements not covered before: the status-line progress indicator, and the Search Details panel (opened from the status-line info icon) with its timings, completion breakdown, and copyable Query ID to share with support when troubleshooting.
  • Corrected the documented query billing unit to 200,000 events.

Tooling:

  • Custom Pygments lexer for LCQL (hooks/lcql_lexer.py) so lcql code blocks are highlighted; registered via the existing MkDocs hooks mechanism and fails safe to plain text (registration is wrapped so any failure degrades to unhighlighted blocks instead of breaking the build).
  • All Python and Go SDK examples, plus the progress-bar helper, extracted to snippets/python/*.py and snippets/golang/*/main.go, embedded with pymdownx.snippets.
  • CLAUDE.md guidance to verify SDK, CLI, and LCQL snippets against authoritative public sources.

CI and security hardening:

  • New workflows: LCQL lexer unit tests; snippet compilation (go build / go vet for Go, py_compile + SDK import check for Python); and a PR docs preview that builds the site and uploads it as a short-lived (1-day) artifact for review. All are read-only, fork-guarded, and never execute snippet code.
  • deploy-pages.yml: permissions scoped to jobs (build gets contents: read; only deploy gets pages: write + id-token: write), with persist-credentials: false and job timeouts.
  • mkdocs.yml: strict: true so build warnings (broken nav, unresolved internal links, bad redirects) fail every build path including deploy, not only the PR preview.
  • Removed docs.yml, a redundant second deploy path that pushed to an unserved gh-pages branch while exposing a write token to the build steps.
  • link-checker.yml: job-scoped permissions, fork guard, the PR internal-link check is now blocking (external stays advisory), the PR-comment step is split into its own minimally-scoped job, the third-party action is SHA-pinned, result artifacts use 1-day retention, and the results-artifact download is continue-on-error so the comment job does not fail spuriously when the build breaks before the link checker runs.
  • snippet-tests.yml: the Python SDK the snippets are checked against is pinned (limacharlie==5.5.1) so the import-path check is reproducible, matching the pinned Go module.
  • publish-release-notes.yml / scripts/publish-release-note.py: sanitize the machine-fed release note before it is written into the docs. The body is HTML-escaped and additionally rejected if any Markdown link/image destination uses a scheme other than http/https/mailto - html.escape does not neutralize those, so a [x](javascript:...) or data: destination would otherwise render as a live one-click link on the docs site. The URL must be https, on an allowlisted host, and free of characters that could break out of the [GitHub Release](URL) Markdown link it is embedded in; component/version use re.fullmatch (no trailing newline); and a length cap plus job timeout bound the work.
  • scripts/check-list-numbering.py (new) + a docs-preview step: fails a PR only on NEW ordered-list numbering breaks - an item the author numbered >1 that renders as a restarted 1. because its continuation content is indented below Python-Markdown's 4-space rule (the Query Console regression fixed in this PR). It is false-positive-free by construction: the author's source numbering is the tie breaker between an accidental split and intentional adjacent lists. Five pre-existing cases are recorded in scripts/list-numbering-baseline.json and are left unchanged here.

Screenshots

LCQL formatting before:

Screenshot 2026-07-09 10 59 11

LCQL formatting after:

Screenshot 2026-07-09 10 58 52

Blast radius / isolation

Documentation content changes are scoped to the Query Console / LCQL section (docs/4-data-queries/). The PR also adds new supporting files (hooks/lcql_lexer.py, snippets/, tests/, scripts/check-list-numbering.py) and makes CI changes: three new workflows, an ordered-list numbering check wired into the PR docs preview, plus hardening of the existing deploy, link-check, and release-note workflows. The docs lexer fails safe (blocks render as plain text if it ever fails to register). Deploy behavior is functionally unchanged - deploy-pages.yml remains the serving path; only its permission scoping is tightened, strict is enforced, and the redundant docs.yml is removed.

How this was verified

  • mkdocs build passes on every build path (now strict) and markdownlint reports 0 errors across all docs.
  • The LCQL lexer unit tests pass (57 tests) and run in CI.
  • All 8 Go snippets compile (go build / go vet) and all 8 Python snippets byte-compile with their SDK import paths verified against the pinned package; both run in CI. The progress-bar Python and Go snippets produce the expected percentage on a sample page.
  • All workflow YAML validates; the release-note publish script passes py_compile, and its URL/body validation is exercised against the XSS and break-out vectors above (rejected) alongside legitimate inputs (accepted).
  • The ordered-list numbering check was validated against the pre-fix Query Console page (caught) and its fixed version (clean) plus adversarial false-positive/true-positive cases, and reports zero new breaks over its baseline.

Follow-up

  • I will update the remaining inline screenshots (time picker, schema fields, table columns) with the new UI elements and information in a separate PR later this week.
  • Settings/ops actions (not code): scope the repository_dispatch token used by the release-notes publisher as tightly as possible; and after this merges, delete the stale gh-pages branch and clear the Pages source-branch setting so the site is served solely from the Actions artifact.
  • Five pre-existing pages have ordered-list numbering breaks recorded in the baseline (sophos, windows/installation, plaso, hayabusa-bigquery, velociraptor-bigquery); each is a mechanical 4-space re-indent and can be cleaned up in a follow-up, dropping its baseline entry.

🤖 Generated with Claude Code

tomaz-lc and others added 11 commits July 9, 2026 10:24
Register a Pygments lexer for the LimaCharlie Query Language so `lcql` fenced code blocks are syntax-highlighted: keywords and operators (including the symbolic `&&` / `||` / `!` forms), string and number literals, field paths, sensor IDs, and aggregation functions.

The lexer is registered through the existing MkDocs hooks mechanism instead of an installed package, and fails safe - if registration ever fails, code blocks fall back to plain text and the build still succeeds.

Add tests that verify the lexer highlights the query language's operator, keyword, and aggregation vocabulary and tokenizes representative queries without error tokens, wired into a CI workflow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g rate

Add a "Query Limits & Performance" page covering query types (stateless, projection, aggregation, stateful) and how each executes (paged vs whole-timeline), data streams, concurrent-query limits, query timeouts, aggregation cardinality guidance, query progress and cost reporting, guidance on writing efficient queries, and troubleshooting.

Correct the documented query billing unit from one million events to 200,000 events, and link the sibling pages to the new reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add a "Time Range" section covering how the range is supplied per interface and the supported relative, absolute, and bounded formats.
- Add a "Common Operators and Patterns" section and expand the stateful operators (`with child` / `with descendant` / `with events`) with sample data; add subtree-wildcard examples.
- Fix correctness issues: use COUNT instead of COUNT_UNIQUE on a GROUP BY key, add projections to reduce returned data, and add high-cardinality GROUP BY guidance.
- Format example queries as `lcql` code blocks and replace sample output with clearly synthetic values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pets

- Document how to discover the per-organization search endpoint via `/v1/orgs/{oid}/url`, with a per-region reference table, and recommend discovery over hardcoding a direct endpoint.
- Use `/` (not `.`) for LCQL field paths in all query examples, and correct the CLI saved-query flags.
- Extract the Python and Go SDK examples into snippets/python/*.py and snippets/golang/*/main.go, embedded with pymdownx.snippets, and add a workflow that compiles them on every push and pull request (go build / go vet for Go, py_compile plus an SDK import-path check for Python) so they cannot drift from the published SDKs.
- The Go examples now use the real SDK methods (NewOrganizationFromClientOptions, org.Query, org.ValidateLCQLQuery, org.EstimateLCQLQueryBilling) and the Hive client.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Instruct contributors to verify SDK, CLI, and LCQL snippets against authoritative public sources (the published SDKs, the CLI, and the LCQL grammar) rather than memory, and point to the compiled snippets in snippets/. Reiterate that the repository is public and only public sources may be referenced.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply least-privilege and untrusted-input hardening to the docs test workflows:

- Set permissions to contents: read (no token write scope; no secrets are used).
- Run pull-request jobs only for same-repository branches, not forks, so untrusted contributors cannot trigger them.
- Set persist-credentials: false on checkout and add per-job timeouts.
- Build Go snippets with -mod=readonly.

The snippets are compiled but never executed (go build / go vet and py_compile do not run the programs), so a maliciously edited snippet cannot execute in CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a pull request from a branch in this repository (collaborators), build the
documentation and upload the rendered site as a downloadable artifact so
reviewers can view the rendered pages before merge.

Build only: no deploy step, no secrets, read-only token, and it does not run for
fork pull requests. A run summary explains how to download and serve the
preview locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s + search details

Replace the Query Console overview screenshot with an annotated version (numbered UI elements) and document two elements not covered before:

- Progress indicator in the status line (how much of the query has completed so far).
- The Search Details panel opened from the status-line info icon, which shows per-session and per-page timings, a completion breakdown, and a copyable Query ID to share with support when troubleshooting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y workflow

Move deploy-pages permissions from workflow scope to job scope: the build job (which runs pip install + mkdocs build, and MkDocs executes local hooks) now holds only contents: read, while pages: write and id-token: write are granted only to the deploy job. Add persist-credentials: false on the build checkout and per-job timeouts.

Delete docs.yml, a redundant second deploy path that pushed to an unserved gh-pages branch while exposing a contents: write token to the build steps. Pages is served from the Actions artifact via deploy-pages.yml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Drop workflow-scope pull-requests: write; use job-scoped permissions.
- Run the build and link-check job only for same-repository pull requests (fork guard), with contents: read and persist-credentials: false.
- Split the PR comment into a separate job that holds pull-requests: write and still posts when the check fails.
- Make the PR internal-link check blocking; keep external-link checking advisory.
- SHA-pin the third-party markdownlint-cli2-action, set the results artifact to 1-day retention, and add per-job timeouts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The release-note url and body are fed via repository_dispatch and rendered by MkDocs with md_in_html enabled. Validate the url (require https and an allowlisted host, dotted-boundary check) and neutralize the body (HTML-escape structural characters, preserving Markdown, with a length cap) before writing it into the docs. Add a job timeout to the publish workflow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tomaz-lc tomaz-lc force-pushed the docs-query-limits-performance branch from 45ee52e to 5077b21 Compare July 9, 2026 10:03
tomaz-lc and others added 7 commits July 9, 2026 15:07
Document how to turn the batch counters into a query progress bar. The
Query Console formula is clamp(batchesCompleted / batchesInScope, 0, 100%),
taking the denominator from the validate response (before running) or from
each page's cumulativeStats (while running) and the numerator from the
per-page cumulativeStats.batchesCompleted, guarding a zero denominator and
clamping the ratio.

Add worked examples in Python, Go, and bash + jq. The Python and Go
examples are extracted to snippets/python/progress_bar.py and
snippets/golang/progress_bar/main.go and embedded via pymdownx.snippets so
they are compiled in CI and cannot drift; both are stdlib-only and operate
on the parsed Search API page response.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
html.escape neutralizes raw HTML but not Markdown link/image destinations,
so a body containing [x](javascript:...) or a reference definition
[x]: javascript:... rendered to a live href - a stored one-click XSS on the
docs site. The url validator was also host-only, so a value like
https://github.com/)[evil](javascript:...) passed the allowlist and then
broke out of the [GitHub Release](URL) Markdown link when embedded verbatim.

Harden the publisher:
- Reject the publish when any Markdown link/image destination in the body
  uses a scheme other than http/https/mailto (fail closed; a machine-fed
  release note never uses javascript:/data:/vbscript:). Raw HTML is still
  HTML-escaped as before.
- Reject a url containing characters that could terminate the Markdown link
  destination or that never appear in a real release url (parentheses,
  angle brackets, quotes, backticks, backslash, whitespace, control chars),
  in addition to the existing https + host-allowlist checks.
- Use re.fullmatch for component/version so a trailing newline can no longer
  be smuggled into the generated heading.
- Replace an em dash in the generated heading with an ASCII hyphen.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lexer registers itself at module import by writing into Pygments'
private _lexer_cache. If that private symbol is ever removed or renamed, the
import would raise and abort the whole mkdocs build - the opposite of the
documented contract that a failure degrades to unhighlighted lcql blocks.
Wrap both the import-time and on_config registration calls in a broad
except so registration failure falls back to plain-text rendering and the
build still succeeds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- mkdocs.yml: set strict: true so build warnings (broken nav, unresolved
  internal links, bad redirects) fail every build path including deploy,
  not just the PR preview.
- link-checker.yml: mark the results-artifact download continue-on-error so
  the comment job no longer fails spuriously when the build breaks before
  lychee runs (the artifact is never created); the comment step already
  no-ops on a missing file.
- snippet-tests.yml: pin the LimaCharlie SDK the Python snippets are checked
  against (limacharlie==5.5.1) so the import-path check is reproducible,
  matching the pinned Go module.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Time Range section referenced non-existent `q` / `qa` CLI subcommands and
implied the CLI carries the time range in the query string. The current
stable CLI exposes `limacharlie search run` and takes the range via the
`--start` / `--end` flags (Unix epoch seconds), like the Search API. Drop the
`q` / `qa` mention, add a CLI row to the interface table, and reword the note
so the leading `-24h |` on the examples is described as raw LCQL rather than
CLI syntax.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The UI Element Overview list indented its nested content by 3 spaces. With
Python-Markdown's 4-space tab length that flattened the sub-lists into the
top-level numbering (so "Sensor Selector" became item 3, etc.) and split the
list into several restarting <ol> blocks, producing the scrambled numbering.

Re-indent all nested content to 4 spaces (8 for content one level deeper) so
the page renders as a single continuous 1-10 ordered list with the sub-lists
under items 2, 4, and 7 correctly nested. Inline text is unchanged; the only
non-whitespace edit is one added blank line before the "Table columns"
sub-item so the preceding image block closes cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a CI check that catches the class of bug where a single authored ordered
list fragments into several restarting <ol> blocks (an item numbered "6."
rendering as a fresh "1.") because its continuation content is indented by
less than Python-Markdown's 4-space tab length. This is exactly the Query
Console UI regression fixed earlier; markdownlint cannot see it because it
aligns to marker width rather than the 4-space rule.

The check is free of false positives by construction: telling an accidental
fragmentation from two intentional adjacent lists is impossible from the
rendered HTML alone, so it uses the author's source numbering as the tie
breaker - a fragment is reported only when its first item was numbered >1 in
the source everywhere it appears (an item the author numbered >1 must never be
the first item of its rendered list). Ambiguous patterns (lists written
entirely with "1." markers, or nested lists that merely flatten while keeping
sequential numbers) are deliberately not flagged.

It runs after the build in the PR docs-preview workflow and fails only on NEW
breaks; the five pre-existing cases are recorded in
scripts/list-numbering-baseline.json and are not touched here. Validated
against the pre-fix Query Console page (caught) and its fixed version (clean),
plus adversarial false-positive/true-positive cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tomaz-lc tomaz-lc marked this pull request as ready for review July 9, 2026 14:35
@tomaz-lc tomaz-lc merged commit 3f331d2 into master Jul 10, 2026
7 checks passed
@tomaz-lc tomaz-lc deleted the docs-query-limits-performance branch July 10, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant